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ABSTRACT 

A study was conducted of the use of outcome 
measurement by the U.S. Department of Health and Human Services (HHS) 
and states to determine whether JOBS (Job Opportunities and Basic 
Skills) program participants are finding employment and leaving 
welfare. In order to assess the progress that HHS has made in 
establishing outcome indicators and goals, researchers from the 
General Accounting Office interviewed officials from HHS and various 
welfare research and interest groups. They also reviewed JOBS 
regulations and reports and surveyed JOBS administrators in the 50 
states and the District of Columbia to determine state performance 
monitoring practices. In addition, they reviewed HHS' proposed 
approach to developing outcome indicators, examined the literature on 
the development of performance monitoring systems, and interviewed 
experts in the field. The study found that HHS does not know whether 
JOBS is reducing welfare dependency because it does not gather enough 
information on critical program outcomes, such as the number of 
participants entering employment and leaving welfare annually. In 
addition, states are held accountable for the number and type of 
participants enrolled in education and training but not for outcomes, 
such as the number of participants finding employment. Although 
little progress has been made in monitoring JOBS outcomes at the 
federal level, nearly all states use some information on participant 
outcomes to manage their individual programs. A majority of states 
monitor the number of JOBS participants entering employment and 
hourly wages at hire. In addition, more than half the states have 
established annual outcome goals. The study recommended that program 
objectives and outcome goals for the JOBS program be clearly defined 
by HHS as soon as possible. (The report of the includes four 
appendixes: scope and methodology of the study; GAO questionnaire 
regarding JOBS program characteristics; comments from HHS; and a list 
of major contributors to the report.) (KG) 
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April 17, 1995 

The Honorable Daniel P. Moynihan 
Ranking Minority Member 
Committee on Finance 
United States Senate 

Dear Senator Moynihan: 

Between 1989 and 1994, federal and state governments spent about 
8 billion dollars on the Job Opportunities and Basic Skills Training (jobs) 
program. The program helps recipients of Aid to Families with Dependent 
Children (afdc) obtain the education, training, and services necessary for 
employment For program managers and policymakers to determine 
whether this investment has helped achieve the objective of reducing 
welfare dependency requires information on jobs participants' outcomes, 
such as whether they are becoming employed and leaving afdc. In working 
toward welfare reform, information on the extent to which jobs is 
achieving its objectives is more important than ever for the Congress. 

. This report responds to your request that we study the use of outcome 
measurement by the Department of Health and Human Services (hhs) and 
states in determining whether jobs participants are finding employment 
and leaving afdc. Specifically, we addressed the following questions: 
(1) What progress has hhs made in measuring the employment and afdc 
status of JOBS participants at the natioiuil level and setting national goals 
against which program performance will be measured? (2) To what extent 
are states measuring participant outcomes and setting performance goals? 
(3) What major issues should be considered in establishing a national 
approach to measuring jobs participant outcomes and setting performance 
goals? 

To assess tlie progress that hhs has made in establishing outcome 
indicators and goals, we interviewed officials from hhs and various welfare 
research and interest groups. We also reviewed jobs regulations and 
reports. Using a mail questionnaire, we surveyed jobs administrators in the 
50 states and the District of Columbia to determine state performance 
monitoring practices. To determine the issues that should be considered in 
developing indicators and goals, we reviewed hhs' proposed approach to 
developing outcome indicators, examined the literature on the 
development of performance monitoring systems, an(i interviewed experts 
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in the field. See appendix I for further details on our scope and 
methodology. 



PpcnltGin Rri pf ™^ know whether jobs is reducing welfare dependency because 

rvchUllo 111 DritJl ^Qgg j^Q^ gather enough information on critical program outcomes, such 

as the number of participants entering employment and leaving afdc 
aimually. In addition, states are held accountable for the number and type 
of participants enrolled in education and training but not for outcomes, 
such as the number of participants finding employment. While the current 
approach to morutoring performance provides important irrformation on 
the activities of jobs participants, state jobs directors are concerned that 
the approach provides little incentive for states to focus on moving 
participants off afdc and into jobs. 



While little progress has been made in monitoring jobs outcomes at the 
federal level, the picture is better at the state level. Nearly all states use 
some information on participant outcomes to manage their ir dividual 
programs, although the extent to which states morutor outcomes varies 
widely. At least in part to demonstrate to their state legislatures that 
program objectives are being achieved, a majority of states monitor the 
number of jobs participants entering employment and hourly wages at 
hire. In addition, over one-half of the states have established annual 
outcome goals. Although many states gather some jobs outcome data, 
without a standard federal approach, few states could provide us with 
comparable data. However, our survey of jobs directors fovmd that 27 
states could provide annual data on individuals entering employment. In 
these states, about 21 percent of jobs participants found jobs in 1993. 

The current national interest in making welfare more employment 
focused, as well as reqvdrements in the Government Performance and 
Results Act (gpra) that performance monitoring become more outcome 
oriented govemmentwide, indicate a need for hhs to move decisively to 
ensure that it meets its current schedule for developing outcome measures 
and goals for jobs, hhs has reported to the Congress that it plans to finalize 
JOBS outcome measures by October 1996 and outcome goals by 
October 1998. A critical first step in developing performance goals will be 
working with the states and other concerned parties to resolve differences 
regarding whether the primary objective of jobs is to help participants 
(1) obtain employment quickly or (2) get the education and training 
needed for better-paying jobs. Congress is considering whether afdc and 
JOBS should be replaced with a welfare-to-work block grant program that 
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includes some jobs' objectives and activities. However this issue is 
resolved, the need for federal accountability would be well served by 
clearly defined program objectives and outcome goals. 



Rflf»k'0rm mH ^^^^^^ Family Support Act (fsa) of 1988, is designed to help 

Da g uui fainilies avoid long-term welfare dependency. The act reqiiires all states to 

establish jobs programs that make available to afdc recipients the 
education, training, and support services they need to prepare for, accept, 
and retain employment. States can provide these services either directly or 
through local service providers. Both the states and the federal 
government share in ti\e costs of the program. States were required to 
begin their jobs programs by October 1, 1990, and be in full statewide 
operation by October 1, 1992. hhs' Administration for Children and 
Families (acf) is responsible for managing jobs at the federal level. 



To understand whether a human services program, such as jobs, is 
achieving its objectives or in need of improvement requires a system for 
gathering information about program performance. Evaluating a program 
by regularly collecting and analyzing performance information is known as 
performance monitoring. Performance monitoring systems include two 
key elements: (1) indicators — ^which define what performance information 
will be gathered and (2) goals — a target level of performance agaiiist 
which actual program performance will be gaug A. 



There are two basic types of indicators: process and outcome. Process 
indicators for jobs would provide information about program activities, 
such as the number of afdc recipient^t participating in jobs, the number of 
JOBS participants receiving training, and the amount of money being spent 
on teenage participants. Outcome indicators for jobs, on the other hand, 
would capture what happens to people after participating in program 
activities, such as the number of people who begin working, the number 
who leave afdc, and the number still employed after 6 months. 



Goals establish the levels of performance that programs are expected to 
achieve. For example, the goal for participants starting work could be 
"25 percent of those participating in jobs each year will enter full-time 
employment." Goals can be established for outcomes, such as the 
percentage of participants finding employment, and processes, such as the 
percentage of participants involved in jobs skills training, for example. 
Goals are often accomparued by financial incentives for meeting or 
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penalties for not meeting goals whether they are related to outcomes or 
processes. 

Because the overall goal of jobs was defined very broadly, states have had 
the flexibility to focus on a variety of different objectives in an effort to 
achieve the broader purpose. These varying objectives can result in 
different approaches for providing jobs services and different program 
results. For example, prograniis with the objective of quickly increasing 
welfare recipients' earnings may emphasize helping participants find any 
job; whereas, pursuing the objective of long-term self-sufficiency may lead 
to more of an emphasis on education and training activities with the hope 
of placing participants in employment that allows them to move off and 
stay off AFDC. If there is no agreement on program objectives, reaching 
agreement on the outcome indicators and goals needed to monitor 
achievement of the objectives will be very difficult. 

FSA mandated that hhs develop outcome goals (known as standards in the 
act) for JOBS outcomes over time and established goals for certain 
processes. It initially reqiaired hhs by October 1993 to recommend jobs 
goals based on specific outcome indicators, such as the number of 
participants who obtained jobs and moved off welfare. This requirement 
was later amended to allow hhs until October 1994 to develop criteria for 
outcome goals for jobs. Through its funding formula, fsa, in effect, set 
minimum goals for two jobs process indicators: late of program 
participation and target group^ expenditures, fsa specified that (1) at a 
minimimi, 20 percent of nonexempt^ adult afdc recipients participate in 
jobs in fiscal year (fy) 1995 and (2) 55 percent of jobs program funds be 
spent in each FY on specified target groups. States are held accotmtable for 
meeting both of these process goals and can lose a portion of their federal 
funding if they fail. 

Recent legislation reinforces the expectation, originally articulated in fsa, 
that HHS develop outcome indicators and goals for jobs, gpra seeks, among 
other objectives, to transform the focus of federal agencies from what they 
are doing to what results they are accomplishing. To accomplish this 



'JOBS target group members include AFDC recipients or applicants who have received AFDC for at 
least 36 months out of the past B years; are under 24 years old and have not completed nor are enrolled 
in high school or have little or no work experience for the preceding year, or ai-e a member of a family 
in which the youngest child Is within 2 years of being ineligible for AFDC because of age. 

^Subject to the availability of state resources, AFDC recipients 16 through 69 years old must participate 
in JOBS unless they are exempt. Reasons for exemption include illness or incapacity, working 30 hours 
or more per week, attending high school, or caring for children under 3 years old. However, teenage 
parents who have not completed high school and have children imder 3 yt irs old are not exempt 



ERIC 



Page 4 GAO/HEHS-9S-86 JOBS Otttcome* 

6 



B-256125 



purpose, the act requires agencies to develop 5 year strategic plans 
beginning in FY 1998 and annual performance plans beginning in FY 1999. 
The strategic plans need to include comprehensive mission statements and 
general goals and objectives for the agencies' msyor functions. The annual 
performance plans, which are based on the strategic plans, should set 
specific performance goals for the year. Performance indicators will then 
be used to monitor progress toward meeting the goals. By adopting a focus 
on outcomes, agency effectiveness and congressional decision-making are 
expected to improve. 

Outcome indicators are usel^ to program managers and policymakers in 
assessing the status of program operations, identifying areas needing 
improvement, and ervsuring accountability for end results. Indicators 
alone, however, do not show the extent to which the program accoimts for 
an observed outcome. For example, suppose 25 percent of jobs 
participants become employed in a certain time period, jobs activities as 
well as events outside the program, such as participants' independent 
efforts to find work or an upsurge in the economy, could account for 
participants finding employment. Determining the extent to which the 
program contributed to the observed outcome involves 'it idles that use 
experimental designs to estimate what would have hapi <iried without the 
program. In this example, to estimate the program's impact, such studies 
might compare the percentage of jobs participants becoming employed 
with the percentage of comparable afdc recipients beconung employed 
without the program. To measure the impact of jobs, fsa authorized 
studies using experimental designs to isolate the actual impact of the 
program. Because such evaluations are usually costly, they are done 
infirequentiy and often involve only select locations. 



HHS Has Made Little 
Progress in 
Establishing an 
Outcome-Focused 
JOBS Performance 
Monitoring System 



Six years after passage of fsa, hhs only holds state jobs programs 
accountable for participation, not employment. As a result, very limited 
national data are available regarding the outcomes of jobs participants. In 
addition, the current approach to performance monitoring does not assist 
states in determining whether they are meeting program goals related to 
employment and independence flrom welfare. According to hhs, a 
combination of technical and environmental factors has impeded the 
development of outcome indicators for jobs. 
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HHS Data Gathering for 
JOBS Focuses on 
Participation, Not 
Outcomes 



HHS focuses its jobs data collection primarily on indicators of participation. 
It collects information from all states on the numbers of program 
participants, expenditures on target group members, and the activities 
individuals are participatii\g in on a monthly basis. States are accountable 
for meeting process goals; if they fail to meet these goals the rates at 
which state expenditures are matched by federal dollars may be reduced.® 

Although HHS has established some outcome indicators, data on these 
indicators present a very incomplete picture of jobs outcomes, hhs data on 
job entry show that on an average monthly basis in fy 1993, 8 percent of 
jobs participants entered employment* In addition to job entry, hhs 
gathers data on hourly wages and whether an individual stopped receiving 
AFDC due to increased income from working. An acf official told us that 
because states are not held accountable for outcomes, neither the states 
nor HHS pays much attention to the monthly outcome data submitted. 

The usefulness of hhs' outcome indicators as tools to nelp manage the 
program is limited for a number of reasons. Because of this approach to 
gathering information, hhs cannot answer important questions regarding 
whether participants are becoming self-sufficient. To measure 
participants' activities, including education, training, job search, and 
employment, hhs gathers data each month on a sample of jobs participants 
who (1) took part in any JOBS-sponsored activity in that month or 
(2) became employed in the sample or preceding month. The sample, 
therefore, excludes anyone who has been employed for more than 2 
months or did not participate in a jobs activity in the sample month. This 
approach to sampling is designed to measure participants' current 
employment-related activities, not outcomes rr'ated to whether 
participants remain employed and move off afdc as a result of their 
earnings. To measure such outcomes, hhs would need to track individual 
participants across time. In addition, current measurement j^proaches do 
not yield armual statistics — a common indicator of program 
performance — on the percentage of jobs participants who became 
employed, hhs also believes that the quality of some of the data is poor. 
For example, an acf official told us that the data on hourly wages are 
uru-eliable because they are missing in many cases and often entered 
incorrectly. 

■•For example, if a state failed to meet the JOBS participation goal of 15 percent of adult nonexempt 
AFDC recipients in FY 1994, the federal matching rate in FY 1995 could be reduced firom 90 percent to 
60 percent (for expenditures up to an amount equal to the ttate't Work Incentive (WIN) program 
allotment for FY 1987). 

*HHS began collecting data on job entry in FY 1992. The raort recent year for which data are available 
is FY 1993. 
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Our 1994 survey found that state jobs directors believe that little progress 
has been made in establishing a performance monitoring system that 
supports achieving program goals related to employment and 
independence from welfare. Eighty-two percent of jobs directors we 
surveyed believe that HHS has not sufficiently moved to focus jobs on 
outcome measurement. Our survey also found that over one-half of state 
JOBS directors believe that the data gathered on participation rates and 
target group expenditures are of little or no use in helping states achieve 
their employment and training program goals. Over one-half of the states 
believed that the nature of federal reporting requirements actually 
hindered their abilities to collect data on outcome indicators. In addition, 
ACF reported in May 1994 that while it spends a significant amount of time 
and resources on monitoring for jobs and other programs, performance 
monitoring at acf is in a "state of crisis," in part because the system does 
not provide a means for determining if programs are producing the desired 
outcomes.^ 

These beliefs echo concerns about the absence of outcome data voiced in 
1989 in response to HHs' notice of proposed rule making (nprm) for jobs. In 
its NPRM for JOBS, HHS Originally included no outcome indicators. Numerous 
commenters on the nprm, however, favored the addition of outcome data. 
Some of them believed that an excessive emphasis on participation would 
drive program operations toward meeting goals not necessarily related to 
achieving independence from welfare through employment. In response to 
the NPRM comments, HHS amended its proposed regulations to include 
some outcome data related to job entry stating that these data should be 
included 

"since employment as a means to self-sufficiency and independence from welfare is the 
objective of the jobs program." 

However, HHS chose not to include additional outcome measures at that 
time, in part, because it agreed with one commenter's concern that 
outcome data do not show the extent to which the program accounts for 
the observed outcomes. 

HHs' lack of an outcome-focused performance monitoring system also 
limits the possibilities for information sharing and coordination with other 
employment and training programs, such as the Job Training Partnership 
Act (jtpa). At the local level, jobs' and jtpa's services are often combined 



^ Report of the Administration for Children and FtoJlies' Monitoring Team , Administration for Children 
and Families, Department of Health and Human Services (May 1994). 
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to meet the education, training, and support service needs of afdc 
recipients. Close coordination is necessary between the programs to 
facilitate effective service delivery. In 1992, the National Governors' 
Association reported that a majority of the jtpa administrators they 
surveyed believed that one barrier to effective coordination was the lack 
of cortsistency between jtpa's outcome goals and jobs' process reporting 
requL'ements.^ 



HHS Has Encountered 
Barriers to Developing an 
Outcome-Focused 
Performance Morutoring 
System 



While FSA set process goals, it also required hhs to develop and submit 
recommendations for outcome goals to the Congress by October 1993. 
These recommendations were to include goals for increased earnings and 
reduced welfare dependency, hhs missed the October 1993 deadline, but 
submitted its report on September 30, 1994.^ This report identified 
problems in developing an outcome-focused performance monitoring 
system and provided a detailed plan and schedule for developing outcome 
indicators and goals. 

In its report to the Congress, hhs identified several technical and 
cn'/ironmental factors that contributed to delays in the issuance of 
recommendations for outcome goals by the October 1993 deadline, hhs 
reported that appropriate outcome indicators had proven difficult to 
define in part due to disagreements among key stakeholders, such as 
reseaichers; congressional staff; and federal, state, and local officials, 
regarding the primary objectives of the jobs program. In addition, setting 
goals was complicated by possible unintended program effects, such as 
programs focusing on the most employment-ready individuals in order to 
meet goals, hhs also reported that turbulence in the welfare system — ^for 
example, funding shortfalls and caseload growth— made it difficult to 
focus the necessary attention and resources on developing outcome goals. 
HHS also wanted to ensure that proposed goals were compatible with 
welfare reform plans being developed by the new administration. 



' JTPA and JOBS: Coordination and Other laaueg , National Governors' AasociaUon (Washington. D.C.; 
October 1992) 

'In legislation passed In late October 1994, the reporting requirement was amended to allow HHS to 
submit a report to the Corip^ss by October 1, 1994, on criteria for the development of outcome goals 
for JOBS. 
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Many States Active in 
Monitoring JOBS 
Program Outcomes, 
but Have Mixed Views 
About Setting 
Nationwide Goals 



In contrast to the relatively slow progress at the federal level, many states 
have been active in developing outcome indicators to monitor jobs 
participant outcomes. To a large extent, this activity has grown out of each 
state program's efforts to demonstrate its effectiveness and gamer support 
for additional state funding. According to our survey results, states use a 
variety of outcome indicators, relying most often on the nimiber of jobs 
participants entering employment and less frequently on job retention rate 
and reductions in afdc payments. Over one-half of the states have also 
established goals for their outcome indicators. Appendix n contains a 
copy of our survey questionnaire and results for selected questions. 

Although many states use their own outcome goals and support 
establishing national goals, they have concerns about how ti\ese goals will 
be set and used. They maintain that hhs may not be able to adequately 
control for differences across states in local economic conditions and 
client characteristics that can affect how successfiil programs are in 
placing participants in jobs. They are also concerned that certain outcome 
indicators will automatically favor certain state programs and unduly 
influence program design decisions, which they believe should be 
maintained at the state level. 



States Use Various 
Indicators to Monitor JOBS 
Outcomes 



Despite the absence of a federal approach to collecting joej outcome data, 
many state programs have been active in tracking jobs outcomes at the 
state level. Our survey of state jobs directors showed that states use a 
variety of indicators to measure outcomes. The two indicators that states 
use most often are the number of jobs participants entering employment 
and their hourly wages at hire (see table 1). 



Table 1: Outcome Indicators Used by 
States 





Number of states using 


Indicator 


indicator* 


Participants entering employment 


49 


Hourly wages at hire 


42 


Participants no longer receiving AFDC due to employment 


33 


Job retention rate 


26 


Participants with reductions in AFDC due to employment 


24 


Educational/training achievement 


24 



*Base Is 50 and Includes 49 states who responded to our survey and the District of Columbia. 
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As sho\/n in figure 1, almost all states reported that they collected data on 
the number of participants entering employment during FY 1993 and 
responded that this is one of the most important indicators to use to 
monitor jobs outcomes. A relatively large number o f states also collected 
data on hourly wages at hire. However, slightly fewcjr states favored using 
hourly wages at hire as an outcome indicator. Several states expressed 
concerns that hourly wages at hire were more a reflection of local 
economic conditions than an outcome of the jobs program. One state 
official also noted that measviring hourly wages at hire could discourage 
programs from placing participants in low-wage entry-level positions, 
which often serve as stepping stones to higher paying positions. 





Indicators Used by the Stales 
Indicators Favored by the States 



1 
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m comparison to employment entry and wages, only two-thirds of states 
collect data on the number of participants no longer receiving afdc due to 
employment. However, as shown in figure 1, a relatively large number of 
states reported that they believe this indicator should be used to monitor 
program outcomes but do not collect the information because of the 
difficulty and resources required to obtain it. 

Three other outcome indicators— job retention rate, the number of 
participants with reductions in afdc due tc employment, and educational 
achievement — are used by almost one-half of the states. Many more states, 
however, favored using these three indicators. While the survey results 
show that state oflBcials believe that job retention rate and changes in afdc 
benefits are particularly useful outcomes to monitor, tracking afdc 
recipients after case closure is difficult, time-consuming, and expensive, 
according to several state officials. Staffing limitations and the inability to 
locate ex-AFDC clients were two reasons they cited for not pursuing these 
indicators more aggressively. Several states also explained that resources 
allocated to data collection are limited and that they are bound to first 
comply with federal reporting requirements. One state official doubted the 
program would be given the authority or resources to gather additional 
data without a federal mandate to do so. 

States also have relied on different approaches to measure outcomes. For 
example, to measure the employment rate of jobs participants, some states 
tracked jobs participants over time and maintained a count of individual 
JOBS participants obtaining or retaining employment over a year. Other 
states performed periodic studies to determine how many jobs 
participants were working and for how long. Other states did no more than 
collect the montWy caseload data required by hhs. 

Given the variation in how states measure employment rates, determining 
the rate at which jobs participants are finding employment on a national 
basis is difficult. However, as part of our survey of state jobs program 
directors, we asked states to provide the number of jobs participants who 
had obtained employment during FY 1993. Based on the responses firom the 
27 states able to provide data in the format requested, approximately 
21 percent of jobs participants entered employment during FY 1993.^ We 
also asked states to provide data on the number of jobs participants 



'Of the 50 states, 27 provided us with total number of JOBS participants and the number who had 
entered employment during the year. These 27 states represent 64 percent of the average monthly 
participants in the JOBS program. Because we do not have a national count of the total number of 
JOBS participants during the year, we could not determine the percent of the total number of 
participants in a year represented by these states. 
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retaiiung their jobs for 3 months. For the nine states reporting this 
information, 33 percent of jobs participants who entered employment 
retained their jobs for at least 3 months.^ 

While states have taken different approaches to measuring outcomes, their 
interest in outcome measurement appears high. As mentioned earlier, 
according to our survey, 82 percent of states indicated hhs has not done 
enough to establish outcome? indicators for jobs. Officials in several states 
emphasized that they need to establish outcome indicators to provide their 
state legislatures with information about jobs participants outcomes. Some 
states were also disappointed that hhs had not introduced outcome 
indicators earlier when states were implementing their jobs data collection 
systems so that they would not have to modify their systems later to meet 
federal reporting requirements. 



Msuority of States Support 
Establishing Outcome 
Goals, but Want 
Substantial Control Over 
Their Development and 
Use 



While states recognize the need and value of outcome indicators in 
managing the jobs program, their views on establishing outcome goals are 
mixed. According to our survey, 29 states had established at least one goal 
at the state level for fiscal year 1993 (see fig. 2). Of these, five states 
reported that they formally adjusted the performance goals to accoimt for 
local differences in client characteristics or the availability of employment. 



•These nine states represent 16 percent of the average monthly number of JOBS participants. 
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Figure 2: States With Outcome Goals 



Connecticut 
Maryland 




state reported establishing at least one outcome goal. 
I I State reported not setting outcome goals, except Iowa which did not respond to survey. 
State reported formally adjusting outcome goals tor local differences. 



As shown in table 2, of the 29 states with established goals, 27 reported 
that they did so for the number of participants entering employment. A 
substantial number of these states also had established goals for hourly 
wages earned at the time of hire, job retention rate, and 
educational/training achievement. 
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Table 2: Outcome Goals Used by 
States 





Number of states with 


Outcome Indicator 


goals* 


Participants entering employment 


27 


Hourly wages at Inire 


16 


Educational/training aclnievement 


14 


Job retention rate 


12 


Participants no longer receiving AFDC due to employment 


9 



"Base is 29 states that reported establishing at least one goal. 



Several states have been particularly active in developing and using 
state-level outcome goals. For example, North Carolina has established 
goals for several outcome indicators, including (1) the percentage of jobs 
participants obtaining employment, (2) afdc closures or reductions due to 
earnings, and (3) the percentage of jobs participants returning to the afdc 
rolls. New Mexico has recently started funding programs based on 
performance; its jobs program will receive state funding based on how 
well it does in meeting established goals. Similarly, California has recently 
imdertaken an initiative to allocate to counties a portion of state funding 
based on performance against designated outcome goals. A recent study 
by the American Public Welfare Association's (apwa) Institute for Family 
Self-Sufficiency also confirmed that more states are establishing outcome 
goals as mechanisms for managing and improving their programs.^" 

A majoriiy of states also support the establishment of nationwide outcome 
goals, although many states are concerned about how goals will be set and 
used at the federal level. Over 90 percent of the states responding to our 
survey indicated that they would like the flexibility to establish their own 
goals or to choose their goals fi-om a menu established by the federal 
govermnent. During follow-up interviews, state officials emphasized that 
they believed certain outcome indicators would favor particular state 
approaches to implementing the jobs program. For example, programs 
that invest more in education and training would benefit from being 
judged on education and training achievement or hourly wages at hire, 
while programs focusing on early initial job search would probably fare 



'"APWA's report, Measuring Client Success: Six States Report on Efforts to Assess What Happens to 
Clients After ThcyRecelve JOBS Services Is based on case studies of Kansas, Maryland, North 
Carolina, Oregon, Pennsylvania, and Wisconsin. APWA's results were similar to GAO's. APWA found 
that each of tJ-ie six states used Job placement as well as a variety of other performance goals to 
manage its programs. In all six states, performance goals were used to publicize the achievements of 
ttie JOBS program, hold contractors accountable to specific goals, and facilitate improvements in 
service delivery. Three states also used goals to determine funding for local programs. 
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better being judged on job entry rate. In either case, states agreed that they 
wanted to retain the flexibility to design their own jobs programs. 

State views were mixed on whether federal funds should be linked to 
meeting national outcome goals. Over 40 percent of the states were against 
linking outcome goals to federal funding, while nearly 30 percent were in 
favor of doing so. Several state officials did not believe that federal 
funding should be tied to national goals because they doubted that hhs 
could sufficiently control for differences in economic conditions and client 
characteristics across various geographical regions. 



Establishing Outcome 
Indicators and Goals 
Is Critical to Making 
JOBS More 
Employment-Focused 



Establishing effective outcome indicators and goals is critical to 
sharpening jobs' focus on the ultimate goals of employment and 
independence from welfare, whether jobs remains the same or is replaced 
with a welfare-to-work block grant program that includes some jobs' 
objectives and activities, as has been proposed. Current congressional and 
public interest in welfare reform as well as gpra requirements indicate a 
need for hhs to move decisively to establish national leadership regarding 
outcome measurement for jobs. Before effective outcome indicators and 
goals can be established, important differences among stakeholders 
regarding the objectives of jobs will have to be resolved. 



HHS Plans to Add 
Outcome Indicators to the 
JOBS Performance 
Monitoring System and 
Modify the System's 
Process Indicators 



In its September 1994 report to the Congress, hhs acknowledged the value 
of and affirmed its commitment to using outcome measurement in its 
performance monitoring system for jobs. Specifically, hhs plans to develop 
outcome indicators and goals and refine existing process indicators and 
goals. In addition, hhs plans to modify the afdc Quality Control system" by 
adding key process indicators, such as participation rates. 

In developing outcome indicators and goals, hhs faces a complex and 
difficult task, hhs recognizes that it must ensure that indicators and goals 
help the program achieve its objectives and meet the needs of numerous 
stakeholders, including local service providers, state and federal 
managers, and policymakers. In its 1994 report to the Congress, hhs 
identified and laid the groundwork for addressing a number of critical 
design and implementation issues that must be addressed to ensure that 
indicators and goals support program objectives, are fair to all states, and 
avoid unintended program consequences. These issues include 

"TTie states and HHS use the Quality Control system to evaluate whether AFDC payments are made 
accurately and to determine how well states comply with regulations to prevent waste, fraud, and 
abuse. 
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(1) developing a process to ensure states are given an equal opportunity to 
meet standards by adjusting for differences among states (that is, levelling 
the playing field), (2) designing strategies to discourage states from 
serving only the easiest-to-serve clients, and (3) selecting data collection 
approaches that are feasible and cost-effective. 

To develop outcome indicators, hhs plans to convene a working group 
composed of representatives from the Congress, hhs, the Department of 
Labor, the states, afdc recipients, commimity-based organizations, and 
others. This group will convene by April 1995 and make recommendations 
to HHS on specific outcome indicators and methods for data collection by 
January 1, 1996. Proposed mdicators are to be published in the Federal 
Register no later than April 1996 and finalized by October 1, 1996. Using a 
similar process, current plans call for outcome goals to be developed and 
finalized by October 1, 1998. 

HHS supports continued use of existing jobs process indicators but believes 
that changes are needed to process goals to make them more effective. In 
addition, in its September 1994 report to the Congress, hhs proposed 
changTig the process goal from the current 20 percent rate to a rate 
between 45 and 55 percent, hhs also suggested changing target-group goals 
to achieve higher levels of participation among the youngest afdc parents. 



Recently, there has been strong congressional and national interest in afdc 
becoming more focused on helping recipients become employed and leave 
AFDC in a limited time period. Nmnerous bills to reform afdc and jobs were 
introduced in the 103rd Congress and more are likely to be introduced in 
the 104th. Several recent welfare reform bills would replace afdc and jobs 
with a welfare-to-work block grant program. To the extent that jobs 
objectives and activities are retained in the block grant, outcome 
indicators and goals for jobs would be useful in ensuring accountability 
and improving congressional oversight. 

While fsa, which originally required hhs to develop outcome goals only for 
JOBS, was limited to one program, gpra requires all federal agencies to 
develop strategic plans, annual performance plans, and annual 
performance reports. Strategic plans articulate the agency's essential 
mission, long-term general goals and objectives, and a plan of action for 
achieving the objectives. The annual performance plans, b: establishing a 
set of performance indicators and goals, provide a link between the 
agency's longer-term objectives and what managers and staff must 
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accomplish on a daily basis to achieve those objectives, gpra requires that 
agencies must submit to Congress &-year strategic plans beginning in fy 
1998 and annual performance plans beginning in fy 1999. The performance 
plaivs are expected to cover each program activity set forth in the agency's 
budget. 

Cvirrently, over 70 agencies and programs are involved in pilot projects for 
GPRA. The National Academy of Public Administration (napa) recently 
reviewed these pilots and found that some agencies had made limited 
progress in developiitg their plans due to insufficient preparation time. 
NAPA recommended that all programs in an agency, not only those 
currently operating as pilots, begin as soon as possible to develop strategic 
and performance plans so that the fy 1998 and fy 1999 deadlines could be 
met. 

If JOBS activities are retained within an afdc block grant, a system of 
accountability for the end results of these activities would still be needed. 
GPRA and the administration's National Performance Review (npr) promote 
the use of outcome indicators for all programs, including those funded 
through block grants. To date, however, data collected by states under 
most block grants have focused on process indicators such as the number 
of clients served. In addition, past block grant programs have not often 
gathered consistent information on program activities and outcomes to 
support congressional oversight. In reviewing data collection under block 
grants, we found that, among other things, national leadership in directing 
the development of model data-gathering criteria could increase data 
comparability and, as a result, oversight. 



Clarifying Program 
Objectives Is First Step in 
Establishing Indicators and 
Goals 



HHS and performance monit-oring system experts agree that the first critical 
step in developing outcome indicators and performance goals is to reach 
agreement among stakeholders, such as the Congress; researchers; and 
federal, state, and local officials, regarding the objectives of the program. 
According to hhs, disj^eements among stakeholders about the objectives 
of the JOBS program have been a major obstacle to developing jobs 
outcome indicators. Difficulty clarifying jobs objectives may again prove 
to be one of the biggest obstacles in the effort to establish outcome 
indicators and goals. 



In our survey of state jobs directors, we found some disagreement 
regarding the programs' overriding objectives. Eighty percent of the 
directors responded that the overriding objective was to prepare and place 
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participants in employment that allows them to move ofif and stay off afdc. 
The other 20 percent stated that their objective was to get paiticipants 
employed in any job, part- or full-time, even if the job might not allow 
them to move off afdc. These two objectives, although consistent with the 
overall objectives of fsa, would likely produce differently designed jobs 
programs and different short-term results, thus making Uie establishment 
of appropriate outcome goals difficult. 



Program managers, policymakers and other stakeholders need to know 
whether jobs participants are finding employment and leaving afdc. Very 
little is known nationally about the outcomes of jobs participants because 
HHS has not moved aggressively on developing an outcome monitoring 
system. Many believe that establishing effective outcome indicators and 
goals is critical to refocusing jobs on *he ultimate goals of employment and 
independence from welfare. Effective indicators and goals would also help 
ensure accountability for achieving these critical outcomes should afdc 
and JOBS be replaced with a welfare-towork block grant program that 
includes some jobs' objectives and activities. The states have a strong 
interest in outcome measurement and are aggressively pursuing a variety 
of approaches in this area State efforts provide a rich well of experience 
that can be drawr* on in developing a national approach to measuring jobs 
outcomes. To meet the requirements of fsa and gpra, hhs must move 
decisively to establish national leadership regarding outcome 
measurement for jobs. 



A cJPn pv romm Pn t<5 ^ March 28, 1995, comments on our draft report, hhs generally agreed 

Agency V_/UmiUeilUs conclusion that its data are incomplete and focused on 

process-oriented goals. However, hhs believes that we did not 
(1) adequately portray the difficulty of developing an outcome-focused 
performance measurement system or (2) give adequate weight to the 
importance of certain technical issues or the progress that hhs has made in 
addressing those issues (see app. HI). We added language in the report to 
more explicitly recognize the difficulty of the task and hhs' progress in 
identifying important technical issues (see p. 15). hhs also suggested minor 
technical revisions to the draft, which we incorporated as appropriate. 
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Our work was performed between January 1994 and February 1995, in 
accordance with generally accepted government auditing standards, with 
the exception that we did not check the accuracy of outcome data 
reported by hhs and the states. 

We are providing copies of this report to the Secretary of Health and 
Human Services, other hhs officials, and state jobs program 
administrators. We will also make copies available to other interested 
parties upon request. Major contributors to this report are listed in 
appendix IV. If you have questions about this report, please call Robert 
MacLafferty on (416) 904-2000. 

Sincerely yours, 




Jane L. Ross 

Director, Income Security Issues 
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Appendix I 



Scope and Methodology 



To determine the progress hhs had made in establishing outcome 
indicators and goals and the issues relevant to their development, we 
interviewed officials from HHS' Administration for Children and Families, 
which is responsible for the jobs program at the federal level. We also 
reviewed (1) the data-reporting procedures for the jobs program, (2) hhs 
reports that siammarize the outcome data collected at the federal level, and 
(3) various reports on the status within acf of monitoring and developing 
outcome measures. We did not verify the accuracy of federal data, but 
were told by HHS officials that the data were not complete or accurate. In 
addition, we reviewed the welfare-to-work and performance measurement 
literature and spoke with officials from various welfare research and 
interest groups, including apwa, the National Governors' Association, and 
the Manpower Demonstration Research Corporation. 

To determine state performance measurement practices, in May 1994, we 
surveyed jobs program administrators in the 50 states and the District of 
Cohmxbia. We also examined 1993 and 1994 reviews of state jobs programs 
conducted by acf's regional offices and obtained available reports on 
outcome data produced by the states. 

In our survey, we used a mail questionnaire to collect information on fy 
1993 general program characteristics, the use of performance indicators 
and goals at the state level, and state preferences regarding the 
development of nationwide indicators and goals. We also requested 
information for selected outcome data elements for fy 1993. We received 
survey responses from the District of Columbia and all 50 states except 
Iowa. However, no respondents could provide complete responses to our 
request for annual, unduplicated outcome data for fy 1993, even though 
many states reported that they monitor some outcome measures. 

To obtain additional information and determine why states did not provide 
requested outcome data, we conducted follow-up telephone interviews 
with officials in 10 states: Alaska, Arkansas, California, Idaho, Louisiana, 
New Jersey, New Mexico, Nevada, New York, and Washington. We first 
identified states that reported that they monitor participants entering 
employment but were unable to provide an annual, tmduplicated count of 
JOBS participants entering employment for fy 1993 (as we had requested in 
our survey). Among these states, we then selected 10 to contact, which 
included states with large, medium, and small caseloads. Based on our 
follow-up work, we determined that six states could not provide annual, 
unduplicated data because the data were not available. In the other four 
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States, the data were available but reporting them would have required 
significant time or resources. 
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Appendix II 

GAO Questionnaire Regarding JOBS 
Program Characteristics 



Our svirvey, sent to state jobs administrators, contained questions on 
general program characteristics, outcome indicators, outcome goals, an< 
participant outcome data. This appendix includes the full text of the 
survey and the aggregate responses for the first three sections of the 
survey. Thy number cited next to each response category is the number of 
states that responsed. 

The appendix does not include the responses regarding participant 
outcome data. Many states were not able to provide the total number of 
individuals served by their programs in fiscal year 1993 and even fewer 
states could answer questions about participant outcomes in the format 
that we requested. As a result, we have not annotated the participant 
outcome data section of the survey with the incomplete data reported to 
us. 
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6A0 Qaestloiuudre Regarding JOBS 
Program Characteristics 



U& GENERAL ACCOUNTING OFFICE 
Survey et JOBS Program Dlrtcton Rtfunliog 
Perfomunct Standardt and Putldpant OotcenM 



INTRODUCTION 

The U. S. General Accounting Office is conducting a study 
of petfoitnance standards and participant outcomes related to 
the Family Support Act of 19S8. SpedHcally, we are 
interested in JOBS performance monitoring practices in your 
state and participant outcome data from your JOBS prograia 
We are asking you to complete this quesaonnaiie as part of 
a survey of all SO state JOBS programs. The survey data 
will be used to provide a national picture of outcomes for 
the JOBS program. We will not use the data to compare 
performance among states. 

INSTRUCTIONS 

This questionnaire should be completed by the person who is 
most knowledgeable about perfonumce monitoring and 
participant outcomes in your JOBS Program. 

Please respond to the following questions for federal fiscal 
year 1993 (October 1, 1992 through September 30, 1993), 
unless otherwise noted. If your records are not organized by 
federal fiscal year, please respond for your sute's fiscal year 

1993. 

Because some terms and their usage may vary across 
institutions, we have provided a glossary of terms that we 
will be tuing in the questionruure. For example, the 
glossary defines outcome data and performance indicators. 
For your convenience, the glossary, listing the temu in 
alphabetical order, is on the inside cover of this 
questionnaire. 



If you have any questtoos about this questioaoaire. please 
call Steve Secrist in our Saa Hsnctsco office, at 
(41S) 904-2236. He vriU be glwl to help you. 

Please return the quettioaaaire in tlie enclosed pce-addiessed 
busineu reply envelope wiAin 14 days of tecdpt If tlie 
envelope is tnisplaoed, please return your queitiomiaiie to: 

Steve Secrist 

U.S. Oeneial Accoundog Office 
301 Howard St.. Suite 1200 
San Frsuicisco, OA 94105 

Before mailing your completed questioimaire, please make • 
copy that you can refer to should we call to tsk for 
additional infonnadoo. 

Thank you for your assistance. 

Please provide the information below about the person 
whom we may contact regarding the completion of diis 
questionnaire. 

Name; 

Title: 

Agency: 

Phone 

Number 
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6A0 QnesttoniMdre Segardiitg JOBS 
Program Charactcriatlcs 



GENERAL PROGRAM CHARACTERISTICS 

I. In FY 93, was your JOBS program state-administered 
or county (locally) administered? (CHECK ONE) 

1. [ 39 ] State-administered 

2. [ U ] County Oocally) administered 



In FY 93, at what point in your JOBS program were 
most participants required to engage in a job search? 
(CHECK ALL THAT APPLY) 

By paitidpant* we mean individuals who participated 
in at least one hour of approved JOBS component 
activities. Approved JOBS activities are those which 
states can provide according to federal regulations. 

1 . ( 5 ) Upon first involvemenf with JOBS 

2. [ 36 ] After completion of formal assessment, if 

determined job ready 

3. [ 18 ] After completion of an education activity, 

but before completion of the participant's 
employability plan 

4. [ 21 ] After completion of a training activity, but 

before completion of the participant's 
employability plan 

5. ( 19 ] After completion of a work-related activity, 

but before completion of the participant's 
employability plan 

6. [25] After completion of jH activities in the 

participant's employability plan 

7. (11] After a determination that an individual is 

not making satisfactory progress in an 
education/training component 

8. I 1 ] Never required 

9. [12] Other (PLEASE SPECIFY) 



10. ( 0 ) Don't know 



In FY 93, had most county (local) JOBS programs in 
your state established a minimum education level at 
which they considered participants ready for 
employment? (CHECK ONE) 

1. (27 ) Yes (CONTINUE) 

2. ( 23 ) No (GO TO QUESTION 5) 

3. (0) Don't know (GO TO QUESTION 5) 

What is the minimum educational level at which the 
largest number of your county (local) JOBS programs 
considered participants ready for employment? 
(CHECK ONE) 

1. ( 10 ] Literacy level at grade 8.9 

2. ( 13 ] High school diploma or GED equivalent 

3. I *] Other (Please specify) 



5. In general, when your state prepared JOBS participants 
for employment in FY 93, which of the following was 
more similar to your program's overriding goal? 
(CHECK ONE) 

1 . ( 10 J To get participants employed m any job, 

part-or full-time, even if the job might not 
allow them to move off AFDC 

2. ( 40 ] To prepare and place participants in 

employment that allows them to move off 
and stay off AFDC 
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6. In most of the county Gocal) JOBS programs in your 
state, what, if anything, happened to mandatory 
participants who refused to participate in a con^uent 
activity in FY W? (CHECK ONE) 

The initial refusal ... 

1 . [ 39 ] started the sanction and reconciliation 

process 

2. [ 6 ] had no immediate consequences, but 

subsequent refusals started the sanction and 
reconciliation process 

3. [ 2 ] had no consequences 

4. [3] Other (PLEASE SPECIFY) 



7. Consider the services your state piovided to JOBS 
participants in FY 93. During the course of providing 
these services, how often, if ever did your state 
comnrnnicate to its JOBS participants that the ultimate 
goal of the JOBS program is employment? (CHECK 
ONE) 

1. [18] Very often 

2. [ 24 ) Often 

3. [ 3 ] Sometimes 

4 [ 1 ) Rarely, if ever 

5 [ 4 ] Don't know/No basts to judge 



8. In FY 93, to what extent, if any, did the JOBS program 
in your state emphasize to JOBS workers that their role 
was to prepare JOBS participants for employment? 
(CHECK ONE) 



1. 


( 15) 


To a very great extent 


2. 


(23) 


To a great extent 


3. 


( 8) 


To a moderate extent 


4. 


( 4) 


To some extent 


5. 


( 0) 


To little or no extent 


6. 


: 0) 


Don't know 



9. In FY 93, to what extent, if any, did the JOBS program 
in your state emphasize to JOBS participants tiuu they 
were obliged to participate in JOBS program activities? 
(CHECK ONE) 



1. 


(21) 


To a very great extent 


2. 


(22) 


To a great extent 


3. 


( 5) 


To a moderate extent 


4. 


( 1) 


To some extent 


5. 


( 1) 


To little or no extent 


6. 


[ 0) 


Don't know 



10. To what extent, if any, has the JOBS program in your 
state moved the welfare system from a focus on 
income maintenance to one promoting work and self- 
sufficiency? (CHECK ONE) 



1. 


( 11 ) 


To a very great extent 


2. 


( 6) 


To a great extent 


3. 


( 17) 


To a moderate extent 


4. 


( 14) 


To some extent 


5. 


( 2) 


To little or no extent 


6. 


( 0) 


Don't know 
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PERFORMANCE INDICATORS 

11. In FY 93, did your state collect data on each of the 
following petfomuuice indlcaton to monitor the 
outcomes of your state's JOBS program? (CHECK 
YES OR NO FOR EACH) 

Pctfomuncc Indicators are tools to monitor client 
outcomes in the JOBS program. By monitoring we 
mean Aat data ate used by managers to assess program 
results. 



PERFORMANCE 
INDICATORS 

1. Educational/training achievement, such 
u graduation rates 

2. Number of participants with reductions 
in AFDC grants (hie to employment 

3. Number of participants no longer 
receiving AFDC due to etnployment 

4. Number of participants entering 
employment 

5. Number of participana entering 
employment dut provides health 
insurance coverage 

6. Hourly wages earned at time of hire 

7. Expected weekly earnings at time of 
hire 

8. Hourly wages earned at a specified 
time period after initial hire 

9. Job retention rate after a specified 
length of time 

10. Rate of return to AFDC, after a 
specified length of time, for those who 
left AFDC due to employment 

11. Other (PLEASE SPECIFY) 



Yes 

(1) 


No 
(2) 


24 


26 


24 


26 


33 


17 


49 


1 


16 


34 


42 


8 


11 


39 


11 


39 


26 


24 


14 


36 


8 


42 



12. What are the most important performance indicators 
that should be used to monitor JOBS participant 
outcomes at the national level? (CHECK ALL THAT 
APPLY) 

1. [ 36 ] Educational/training achievement, such as 

graduation rates 

2. [ 39 ] Number of participants with reductions in 

AFDC grants due to employment 

3. [ 44 ] Number of participants no longer receiving 

AFDC due to employment 

4. [ 46 ] Number of participants entering 

employment 

5. [ 29 ] Number of participants entering 

etnployment that provides health insurance 
coverage 

6. [ 35 ] Hourly wages earned at time of hire 

7. [ 13 ] Expected weekly earnings at time of hire 

8. [ 20 ] Hourly wages eanied at a specified time 

period after initial hire 

9. [ 40 ] Job retention rate after a specified length of 

time 

10. [ 41 ] Rate of return to AFDC, after a specified 

length of time, for those who left AFDC 
due to employment 

11. [7] Other (PLEASE SPECIFY) 
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6A0 Qneitionnalre Regarding JOBS 
Program Characteristics 



17. Which of the following performaoce indicators, if any, 
does your state believe should not be used to monitor 
JOBS participant outcomes at the national level? 
(CHECK ALL THAT APPLY) 

1. [ S ] Educational/training achievement, such as 
graduation rates 



14. For each of the performance indicators thnt you 
checked in quesiicn 1 3, pleass briefly describe the 
reason why your state believes it should not. be used to 
monitor JOBS participant outcomes at the national 
level. 



2. [ 4 ] Number of participants with reductions in 

AFDC grants due to employment 

3. [ 2 ] Number of participants no longer receiving 

AFDC due to employment 

4. [ 1 ] Number of participants entering 

employment 



5. ( 9 ] Number of participants entering 

employment that provides health insurance 
coverage 

6. [ 9 ] Hourly wages earned at time of hire 

7. [ 18 ] Expected weekly earnings at time of hire 

8 [ IS ] Hourly wages earned at a specified time 
period after initial hire 

9. [ 4 ] Job retention rate after a specified length of 

time 

10. ( 3 ] Rate of return to AFDC, after a specified 

length of time, for those who left AFDC 
due to employment 

11. [3] Other (PLEASE SPECIFY) 



IS. Which one of the following methods for establishing 
nationwide JOBS performance indicators would your 
state prefer? (CHECK ONE) 

1 . [ 0 ] Federal government, in consultation with 

the sutes, establishes the indicators for all 
states 

2. [ 34 ] States have flexibility to choose indicators 

from a menu established by the federal 
goveniment in consultation with the states 

3. [ 12 ] States have complete flexibility to establish 

their own indicators 



4. [4] Other (PLEASE SPECIFY) 
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6AO Qnestloiiaalre Regarding JOBS 
Program Characteriatic* 



16. How much do you tfaiok HHS' Administration for 
Childien ind Funilies (ACF) has moved the JOBS 
program to focus on peifonnance indicators related to 
participant outcomes- mote dian enough, enough, or 
less than enough? (CHECK ONE) 



1. 


[0] 


Much more than enough 


2. 


[ 1 ] 


More than enough 


3. 


[ 7] 


Enough 


4. 


[25] 


Less than enough 


5. 


[16] 


Much less than enough 




[ 1 ] 


No answer provided 



17. During FY 93, how easy or difficult was it for your 
state to collect data on outcome-related performance 



indicators? 


(CHECK ONE) 


1. 


[ 0] 


Very easy 


2. 


[ 5] 


Somewhat easy 


3. 


[ 5] 


Neither easy nor difTicult 


4. 


[23] 


Somewhat difBcult 


5. 


[17] 


Very difficult 


6. 


[ 0] 


No basis to judge 



18. During FY 93, to what extent, if any, did each of the following factors hinder your program's ability to collect daU on 
outcome-related performance indicators? (CHECK ONE FOR EACH REASON) 



PERFORMANCE INDICATORS 


Little or No 
Extent 

(1) 


Some 
Extent 

(2) 


Moderate 
Extent 
(3) 


Great 
Extent 
(4) 


Very 
(jreat 
Extent 
(5) 


1. Availability of funds 


6 


16 


10 


13 


5 


2. Availability of staff 


3 


6 


9 


20 


12 


3. Level of automation and data 
systems 


5 


9 


10 


10 


16 


4. Nature of current federal data 
gathering requirements 


5 


3 


15 


12 


15 


3. Amount of guidance provided 
by ACF on JOBS 
performance indicators 
related to outcomes 


18 


6 


12 


7 


7 


6. Others (PLEASE SPECIFY) 
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PERFORMANCE STANDARDS 

19. In FY 93, did your JOBS program have sate- wide 
pcrfomiuicc standards for any of the outcomes you 
monitor? 

Performance tUndank are benchmarki for given 
performance indicators. For ewunple, a program may 
collect data on the performance indicator "number of 
participants entering employment" Its performance 
standard may be "25 percent of participants will gain 
full-time employment each year." (CHECK ONE) 

1. [29] Yes (CONTflWE) 

2. [ 21 ) No (GO TO QUESTION 22) 

50. For which JOBS participant outcomes did your state 
have state-wide performance standards in FY 937 
(CHECK ALL THAT APPLY) 

1. [ 14 ] Educational/training achievement, such as 

graduation rates 

2. [ 3 ) Number of participants with reductions in 

AFDC grants due to employment 

3. [ 9 ) Number of participants no longer receiving 

AFDC due to employment 

4. [ 27 ) Number of participants entering 

employment 

5. [ 3 ) Number of participants entering 

employment that provides health insurance 
coverage 

6. [ 16 ] Hourly wages eariKd at time of hire 

7. [ 2 ) Expected weekly earnings at time of hire 

8. [ 1 ] Hourly wages carried at a specified time 

period after initial hire 

9. [ 2 ] Job retention rate after a specified length of 

time 



21 . In FY 93, did your state make any formal adjustments 
in its performance standards to account for local 
differences in either client characteristics or the 
availabiUty of jobs? (CHECK ONE) 

1. [ 5 1 Yes 

2. [ 24 ) No 



22 In FY 93, how many county (local) JOBS programs in 
your state had established countywide performance 
standards related to participant outcomes? (CHECK 
ONE) 

1. [ 16 ) All 

2. [2] Most 

3. [0] About half 

4. [7] Some 

5. [ 25 ) None 



23. In FY 93, how many counties in your state had 

established performance standards related to participant 
outcomes for county (local) JOBS program managers 
or staff ? (CHECK ONE) 



1. 


[ 7) 


All 


2. 


[ 31 


Most 


3. 


I 0) 


About half 


4. 


[ 3) 


Some 


5. 


[30] 


None 




[ I 


) No answer provided 



10. [ 3 ) Rate of return to AFDC, after a specified 

length of time, for those who left AFDC 
due to employment 

11. [01 Other (PLEASE SPECIFY) 
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24. 


In your view, should nationwide performance standards 


27. 


Consider the data you were required to collect and 




related to participant outcomes be established for the 




report to meet federal repotting requirements on JOBS 




JOBS program? (CHECK ONE) 




parttdpstion rates in FY 93. How easy or difficult 




1. [ 9 ] Definitely yes 




was it for your state to gather the data required to meet 






these reporting requirements? (CHECK ONE) 




2. [ 17 ] Probably yes 




1. ( 


0 ) Very easy 




3. [ 8 ] As much yes as no (undecided) 




2. ( 


3 ] Somewhat easy 




4. [ 10 1 Probably no 




3. [ 


2 ] Neither easy nor difficult 




5. I 6] Definitely no 




4. ( 


24] Somewhat difficult 


25. 


If the federal Rovemtra,nt established nationwide 




5. [ 


21 ] Very difficult 




performance standards related to participant outcomes 








for JOBS, which one of the following methods for 




6. ( 


0 ] No opinion 




establishing standards would your state prefer? 








(CHECK ONE) 


28. 


When 


your state collects federally required JOBS 








participation rate data, how useful, if at all, ate these 




1. [ 1 ] Federal government, in consultation with 




data in helping your sute achieve its JOBS 




the sutes. establishes the standards for all 




employment and training goals? (CHECK ONE) 




states 














1 ( 


1 ] Very useful 




2. [ 33 ] States have flexibility to choose standards 










from a menu established by the federal 




2- ( 


2 ] Moderately useful 




government in consultation with the states 














3. [ 


20 ] Somewhat useful 




3. [ 13 ] States have complete flexibility to establish 










their own standards 




4. ( 


26 ] Of little or no use 




4. 13 1 Other (PLEASE SPECIri) 




5. ( 


1 ] No Opinion 






29. 


Consider the data you were required to collect and 


26. 


If the federal Eovemment established nationwide 




report to meet federal repotting requirements on JOBS 




performance standards related to participant outcomes 




Urget groupi in FY 93. How easy or difficult was it 




for JOBS, should amounts of federal funding to state 




for your state to gather the data required to meet these 




JOBS programs be linked to meeting these standards? 




reporting requirements? (CHECK ONE) 




(CHECK ONE) 














1. 


3 ) Very easy 




1. [ 3 ] Definitely yes 














2. 


9 ] Somewhat easy 




2. ( 11 ) Probably yes 












3. 


9 ] Neither easy nor difficult 




3. [ IS ] As much yes as no (undecided) 












4. 


18 ] Somewhat difficult 




4. [ 8 ] Probably no 














5. 


10] Very difficult 




5. [ 13 ] Definitely no 












6. 


1 ] No opinion 




ft. [ 0 ] No opinion 
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30. When your state coUecti federally required data on 
JOBS target groups, how useful, if at all, are these 
data in helping your state achieve its JOBS 
employment and training goals? (CHECK ONE) 

! [ 2 ) Very useful 

2. [ 5 ] Moderately useful 

3. [ 12 ] Somewhat useful 

4. [ 30 ] Of little or do use 

5. [ 1 ] No Opinion 



JOBS PARTICIPANT OUTCOME DATA 

In this section of the questionnaire we are interested in data 
that are reprttcntative of yoor entire state, if your state 
collects participant outcome data on a sample basis, please 
generalize to the entire JOBS caseload in answering the 
participant outcome questions. If you do not believe your 
sample data are representative of the entire state, please 
check the "data are not available" box. 

31. Are you providing information in this section of the 
questionnaire by federal fiical year or state fiscal year? 
(CHECK ONE) 

1. [ ) Federal fiscal year (10/1/92 - 9/301^93) 

2. [ ] State fiscal year -> What is the beginning 

and ending date for your state's fiscal year 
1993? 



32. 



to 

Month/Year Month/Year 



Of your total AFDC and AFDC-UP recipients in FV 
93, how many participated for at least one hour, at any 
time during the year, in an approved JOBS activity? 
An approved JOBS activity is one which states can 
provide according to federal regulations. (ENTER 
UNDUPLICATED NUMBER) 



Number participating in an approved 
JOBS activity 



[ ) Datt are not avaiIable--> (GO TO QUESTION 
51 ON PAGE 12) 



Employment Entry and Retention 



33. 



32 



Of diose in an approved JOBS activity in your state in 
FY 93, how many entered subsidized or unsubsidizcd 
employment at some point during the year? (ENTER 
NUMBER; IF NONE, ENTER '0') 



1. 



34. 



Entered subsidized etnployment 



[ ] Data are not available 



Entered unsubsidized etnployment 



[ ] Data are not availabIe-> 



(GOTO 
QUESTION 
46 ON 
PAGE 11) 



Of tiiose who entered unsubsidized employment in FY 
93, how many had high school diplomas or GEDs? 
(ENTER NUMBER; IF NONE, ENTER 'O') 

Had high school diplomas or GEDs 



35. 



[ ] Data are not available 



Of diose who did not enter unsubsidized employment 
in FY 93, how many had high school diplomas or 
GEDs? (ENTER NUMBER; IF NONE, ENTER 'O') 



. Had high school diplomas or GEDs 



36. 



[ ] Data are not available 



Of tiiose who entered unsubsidized employment in FY 
93, how many had any work experience in tiie previous 
12 montiis? (ENTER NUMBER; IF NONE, ENTER 
•0') 



. Had work experience in die previous 12 
montiis 



[ ] Data are not available 
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37. 



Of those who did not enter unsubsidized employment 
in FY 93, how many had any work experience in the 
previous 12 months? (ENTER NUMBER; IF NONE, 
ENTER '0') 



. Had work experience in the previous 12 
montlu 



42. 



38. 



[ ] Dau are not available 

Of those who entered unsubsidized employment in FY 
93, how many, if any, took (1) full time jobs (working 
30 hours or mote per week) and (2) part-time jobs 
(working less than 30 hours per week)? (ENTER 
NUMBERS; EF NONE, ENTER '0') 



1. . 



Took full-time jobs 



[ ] Data are not available 

2. Took part-lime jobs 

! ] Data are not available 

39. Of those who entered unsubsidized employment in FY 
93, how many took jobs which provided them with 
health insurance? (ENTER NUMBER; EF NONE, 
ENTER '0') 

Took jobs with health insurance 



[ ] Data are not available 

40. For those who entered unsubsidized employment in FY 
93, on average how many months were they in an 
approved JOBS activity prior to entering employment? 
(ENTER NUMBER; IF NONE ENTER '0') 

Months in an approved JOBS activity 



( ] Data are not available 

41 . Consider the individuals in an approved JOBS activity 
who entered unsubsidized employment in your itate 
during FY 93. Were follow-up efforts made to 
determine If they are itill employed? (CHECK ONE) 

1. ( ) Yei (CONTINUE) 

2. ( ) No (GO TO QUESTION 43) 



Consider again individuals in an Eq)proved JOBS 
activity who entered unsubsidized employment during 
FY 93. Please indicate the time period(s) during which 
you followed-up on these individuals, and for each 
applicable time period, provide the number of 
individuals who remained employed. (CHECK YES 
OR NO FOR EACH; EF YES, ENTER NUMBER) 





YES 
(1) 


NO I 
(2) J 


1. After 3 
months 






2. After 6 
months 






3. After other 
titne period 
(SPECIFY) 







NUMBER 
REMAINED 



Change* in AFDC RKdpt for Thote Who Entered 
Employment 

43. For those in an approved JOBS activity who entered 
unsubsidized employment in FY 93, how many were in 
the following categories at time of job entry? (ENTER 
NUMBERS; IF NONE, ENTER 'O') 

1. Remained on AFDC 

with no gnnt reduction 



[ ] Date are not available 

2. Remained on AFDC 
with a reduced cuh grant 

[ ] Data are not available 

3. Stopped receiving 
a cash grant 



( ] Dau are not avaiUble ~> (GO TO QUESTION 
46) 
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44. Consider the individutk in an approved JOBS activity 
who stopped receiving AFDC due to employment in 
FY 93. Were follow-up efforts made to determine if 
they remained off AFDC? (CHECK ONE) 

1. [ ] Yes (CONTINUE) 

2. (1 No (GO TO QUESTION 4«) 



45. Consider again the individuals in an approved JOBS 
activity who stopped receiving AFDC due to 
employment in FY 93. Please indicate the time period 
during which you followed-up on these individuals, and 
for each applicable time period, provide the number of 
individuals who remained off AFDC due to 
employment. (CHECK YES OR NO FOR EACH; IF 
YES, ENTER NUMBER) 



1. After 3 months 



After 6 months 



period 
(SPECIFY) 



YES 
(I) 


NO 1 
(2) J 















NUMBER 
REMAINED OFF 



Educmtkm Component 

46. Of those in an approved JOBS activity in your state 
during FY 93, how many participated in an education 
component, that v ::d'.''> basic education, English-as-a- 
Second Langiuge, high school, GED, or post-secondary 
education? (ENTER UNDUPUCATED NUMBER; 
IF NONE, ENTER '0') 



Number in an education 
component 



47. Of those in your state's JOBS program who 

participated in an education component, how many 
successfully completed that component in FY 93? 
(ENTER NUMBER; IF NONE, ENTER '0') 



Successfully completed education 
con^oem 



[ ) Data are not available 



48. Of those in your stale's JOBS program who 

successfully completed an education component during 
FY 93, how many entered unsubsidized employitKnt in 
FY 93? (ENTER NUMBER; IF NONE. ENTER '0') 

Entered unsubsidired employment 



[ ) Data are not available 



49. Of those in your stale's JOBS program in FY 93 who 
successfully completed an education component, how 
many did each of the following. (ENTER NUMBERS; 
IF NONE, ENTER '0') 

1 . Obtained high school diploma 

or GED 



[ ) E>ala are not available 
Obtained post-secondary degree 
[ I Dal* are not available 
Other (PLEASE SPECIFY) 



[ ] Dal* are not available-> (GO TO 

QUESTION 51) 



50. Did your slate test those who completed an educational 
component in FY 93 to determine whether they made 
any measurable educational gains as a rcilt of their 
participation? (CHECK ONE) 

1. [ ] Yes 

2. [ 1 No 
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Tnining Ccsst'oncnt 


Tetn Pmrtnti 


51. 


Of those in an approveti JOBS activity in your state 


55. 


How many teen parents participated in an approved 




during FY 93, how many participated in a training 




BCuViiy ID your siwc s j\joo piugioni Qunng r i yj: 




component (program), that is, one which provides 




(ENTER NUMBER; IF NONE, ENTER '0') 




vocational training in technical job skills and/or helps 








develop knowledge and abilities in a specific 




Teen parents in approved JOBS 




occupational area? (ENTER UNDUPLICATBD 




activity 




NUMBER; IF NONE, ENTER '0') 












[ ] Data are not a"<ilable -> (GO TO 




Number in a training component 




QUESTION <0) 




[ ] Data are not available -> (GO TO QUESTION 


56. 


Of the teen parents participating in an approved JOBS 




55) 




activity, during FY 93 how many were in a high 








school, GEO, basic education, or similar education 


52. 


Of those in your state's JOBS program who were in a 




program? (ENTER NUMBER; IF NONE, ENTER '0') 




training component during FY 93, how many 




Teen parents in education prosramt 




successfully completed the training component in FY 






93? (ENTER NUMBER; IF NONE, ENTER 'O') 












[ ] Data are not available 




Successfiillv completed training 








component 




Of the teen parent* i'. an education program during FY 






57. 




[ ] Data are not available 




93. how many cotripleted their high school degree or 








equivaltnt educat'on program in FY 93? (ENTER 








NUMBER; IF NONE. ENTER '0') 


53. 


Of those in your state's JOBS program who 








successfully completed a training component during FY 




Tefti parents comnleted hieh 




93, how inany entered unsubsidizcd employment in FY 








93? (ENTER NUMBER; IF NONE, ENTER '0') 












[ ] Data are not available 




F.ntered unsubsidized employment 








( ] Data are not available 


58. 


Of those teen parents who completed a high school 








degree or equivalent education program, how many 








entered employment in FY 93? (ENTER NUMBER; IF 


54. 


Of those in your state's JOBS program who entered 




NONE ENTER '0') 




unsubsidized employment in FY 93 after completing a 








training component, how many, if any, entered an 




Teen parents who comoleted a 




occupation for which they had been trained by the 




high school degree or etpiivalent 




JOBS program? (ENTER NUMBER; IF NONE. 




and entered e' iployment 




ENTER '0') 












[ ] Data are not available 




Entered occupation for which thev were 








trained 








[ ] Data are not available 
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59. 


Of those teen oaients without a hizb school detree or 


M What wu ttv tnfil anmi*! rAtt (ttAr%\ and ttab*^ nf 




equivalent, how many enteied employment in FY 937 


your JOBS program during FY 93? Pleaie exclude 




(ENTER NUMBER; IF NONE, ENTER 'O') 


child care costs and services obtained ftom odier 






programs and sot charged to (be JOBS budget 




Teen parents \vi4>9<it * 


(ENTER AMOUNT) 




school degree or GED and enteted 






employment 


t 




[ ] Data are not available 


[ ] Dau are not available 






0^. nave me siaiewiue *\jd^ pcognuu, a componeiu oi mc 


PARTICIPATION, COST AND. PROGRAM 


statewide program, or a county (local) program ever 


EVALUATIONS 


been die subject of a program evaluation which used an 






experimental design to compare die outcomes of those 


60. 


Of your total AFDC and AFDC-UP recipients in 


who participate in JOBS wiA die outcomes of those 




January, 1994, how u>any participated in an approved 


who do not participate in JOBS. (CHECK ONE) 




JOBS activity during the month? (ENTER 






UNDUPUCATED NUMBER; IF NONE, ENTER '0') 








1. [ ] Yes -> Please provide die contact penoo. 






name tmd telephone number of 




Numher participating in as 


die organization conductinf die 




approved JOBS activity 


evahiatioft. 






Contact 


61. 


Of those in an approved activity in your state's JOBS 


person: 




program in January, 1994, how many had participated 






in JOBS for a total of 2 yean or more? (B^ER 


Name of 




NUMBER; IF NONE, ENTER 'O'.) 


nrguiiutinn- 




Number participating in JOBS for 






2 years or more 








Telephone 




[ ] Data are not available 


Numher 






2. [ ] No 


62. 


What is the average length of stay in montht for a 






single spell on AFDC, for all AFDC recipients in your 






state and for Aote in an approved JOBS activity? 






(ENTER NUMBERS) 






I . Mondis for all AFDC recipienU 






[ ] Dau are not available 






2. Months for those in an approved 






JOBS activity 






[ ] Data are not available 
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65. In your opinion, which county (or other local 
jurisdiction) within your state operates the most 
effective JOBS program? By effective we mean a 
program that achieves state program goals and can 
document its success with outcome-based perfonnance 
data or program evaluation results? 

Name of local jurisdiction; 



COMMENTS 

67. In this questionnaire, we ask about statewide 

perfonnance data related to employment, job retention, 
AFDC status, and education and ttaining. Have you 
collected any other data on JOBS outcomes in your 
state-- for example, from a qjecial progratn, a pilot 
program, or a demonstration project? (CHECK ONE) 



Please provide the name and phone niunber of the 
county JOBS director for the county named above. 



1. [ ] 



2. [ ] 



Yes 



No 



-> PLEASE PROVIDE YOUR 
DATA 



Name 



Phone Number (_ 



68. Please provide below any comments that you might 
have about this questionnaire, perfonnance indicatora 
and standards, or client outcomes in the JOBS program. 



66. In the space below, please briefly describe your 
reason(s) for selecting the above program. 
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GLOSSARY 



AFDC RedpleDt - Ai used in this questiomiaire, AFDC 
recipient tefera to the parent or parents teceiving AFDC 
beaefiti lod does not include dqiendent children. 

Approved JOBS Activity -- A JOBS activity which can be 
provided by stales according to federal regulations. 

Education - Education includes programs such as adult 
basir. education, English-as-a-Second Language, high school, 
GED, and post-secondaiy degrees. 

Outcome Data -- Isfonnation about ttie sutus of individuals 
enrolled in JOBS at a given point in time. Examples of 
outcome data include the number of individuals who find a 
job, the number of individuals who are still employed after 1 
year, and the number of individuals who leave AFDC due to 
wages. 

Parttdpantf -- Individuals who participate in at least one 
hour of approved JOBS component activities. 

Performance Indicator - A performance indicator is a tool 
to monitor clirnt outcomes in tlie JOBS program. For 
example, one possible petfonnance indicator for the JOBS 
program could be the percent of those enrolled in the 
program who Cod jobs annually. 

Performance Standard - A performance standard is a 
benchmaric or quaniitalive target for a given performance 
indicator. For example, 75 percent of teen parents enrolled 
in JOBS each year will complete their high school 
education. 



Subaidixed Employment Subsidized employment requires 
full or partial payment of the wages frixn government ftmds. 
Examples of this include on-die-job training and woik 
supplementation. 

Training - Training programs provide vocational training in 
technical job skills and/or help develop knowledge and 
abilities in a specific occupational area. 

Unrabddized cmiiloyiBeBt ~ In uniubsidized enploymeot, 
the wages of die employee are paid solely by the employer. 
For the purposes of this questionnaite, this includes 
employment for which an employer receives the Targeted 
Jobs Tax Credit. 
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DErARTMENT OF HEALTH h HUMAN SERVICES 



Offlo* ol Intpwtor o«n«nt 



WMhlngton, O.C. 20201 



MAR 28 1996 



Ms, Jane L. Ross 
Director, Income 

Security Issues 
United States General 

Accounting Office 
Washington, D.C. 20548 

Dear Ms. Ross-. 

Enclosed are the Department's comments on your draft report, 
"Welfare to work; HHS Does Not Know If JOBS Participants Are 
Becoming Self -Sufficient . " The comments represent the tentative 
position of the Department and are subject to reevaluatlon when 
the final version of this report is received. 

The Department appreciates the opportunity to comnent on this 
draft report before its publication. 

Sincerely yours. 




June Olbbs Brown 
Inspector General 



Enclosure 
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Thanlc you for the opportunity to comnent on the draft report. 
General rnmmoni-H 

The report identified many of the problems with data collection, 
the lack of consensuB by Job Opportunities and Basic Skills 
Training (JOBS) directors on outcome goals and maasures, and 
whether such goals and measures should be memdatory and/or 
subject to penalty. We agree that our data is incon?)lete and 
that our data collection has been largely focused on orocess- 
oriented goals. 

We believe the report minimizes both the legislation's role in 
establishing the process measures and the States' inability to 
accurately report even the current required data elements. As 
the Department's Report to Congress indicated, there is general 
a3ret\ment that there should be outcome measures. Deciding on 
effective, equitable outcome goals and measures which allow state 
flexioility and account for variations across States is not easy. 
The only outcome measure that 49 of the 50 States presently 
collect is one that is currently required by the Department 
(i.e., JOBS participants entering employment). 

Outcome goals drive program operations. With the interdependence 
of welfare-related programs like Aid to Families with Dependent 
Children (AFDC) , Pood Stan^js and the Job Training Partnership 
Act, outcomes need to be consistent among related programs. 

We have identified several issues that need to be addressed and 
dealt with prior to using outcomes as the basis for performance, 
measurement and standards. These issues include: 

an inconsistent relationship between outcomes and 
program effectiveness; 

the establishment of a "level playing field" across and 
within States, and over time; 

the determination of who is "counted" in measuring, 
performance; and 

the recognition that different state JOBS programs may 
have different objectives. 

While these issues are briefly discussed by QAO, we do not 
believe that their importance nor the progress the Department has 
made in addressing these issues is given adequate weight, in 
particular, the Department has worked closely with researchers. 
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academics, and Federal, State, and local oificials to identify 
and develop methods to resolve these issues. This work has been 
critical in working towards the development of a performance- 
based system which will be effective in promoting both high 
performance programs and accovmtability. The Report to Congress 
prepared by the Department and submitted in Septeinber 1994, 
provides more detail on these issues and activities. The GAO 
report would be more balanced, accurate, and useful if it 
reflected both the issues faced in developing a responsible 
performance measurement system and the strides the Department has 
made in overcoming them. 

Additionally, we believe that the title on your draft report does 
not accurately portray its content. We suggest that you change 
the title of your report to: "JOBS Outcome Indicators and 
Performance Goals." 
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Stephen D. Secrist, Evaluator-in-Charge, (415) 904-2000 
Susan E. Arnold 
Kay E. Brown 
LuArm M. Moy 
AnnT. Walker 
Christina L. Warren 
Karen D. Wright 
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Ordt'riii};? Information 

The first copj- of each G AO report and testimony is free. 
Additional copies are $2 each. Orders should be sent to the 
following address, accompanied by a check or money order 
made out to the Superintendent of Documents, when 
necessary. Orders for 100 or more copies to be mailed to a 
single address are discounted 25 percent. 

Orders by mail: 



U.S. General Accounting Office 
P.O. Box 601.') 

Gaithersbiu-g, MI) 20884-6015 ' 



or visit: 



Room 1100 

700 4th St. N\V (corner of 4th and G Sts. NW) 
U.S. General Accounting Office 
Washington, DC - 



Orders may also be^placed by calling (202) 512-6000 

or by using fax number i'iOl) 258-4066, or TDD (.301) 413-0a0(;. 

Each day, GAO'issues a list of newly available reports and 
testimony. .To receive facsimile copies of the daily list or any 
list from the pjust .'JO days, please call (.'K)l) 258-4097 using a 
touchtone phone. A recorded menu will provide information on 
how to obtain these lists. 
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